Enable soft delete for workspace backup Recovery Services vaults and fix core public IP lifecycle issues by Copilot · Pull Request #4907 · microsoft/AzureTRE

Copilot · 2026-05-21T15:07:57Z

What is being addressed

Workspace creation with backup enabled can fail when the workspace backup Recovery Services Vault is created with soft delete disabled. Azure now requires soft delete to be enabled for new vaults, so attempts to create the vault with soft_delete_enabled = false can be rejected and leave the workspace deployment stuck.

This change also needs to preserve current workspace delete behavior so backup-related resources do not block cleanup.

Additionally, Azure automatically sets ip_tags on public IPs (e.g. "FirstPartyUsage": "/Unprivileged"), which causes Terraform to force replacement of the firewall transit public IP and the application gateway public IP on subsequent applies. This cascades into unnecessary recreation of dependent resources such as private DNS zones.

How is this addressed

Recovery Services Vault configuration
- Keep the workspace backup vault on azurerm_recovery_services_vault
- Remove the deprecated soft_delete_enabled argument (removed in AzureRM v4.64.0+); soft delete is now enabled by default on the platform
- Preserve the existing Zone Redundant storage configuration
- Preserve the existing CMK configuration on the AzureRM vault resource
Delete behavior
- Configure the workspace base AzureRM provider recovery_service features to purge protected items from the vault on destroy
- Set vm_backup_stop_protection_and_retain_data_on_destroy = false so workspace cleanup continues to remove backup data rather than retaining it
Core infrastructure — public IP lifecycle fixes
- Add ip_tags to lifecycle { ignore_changes = [...] } on azurerm_public_ip.fwtransit in core/terraform/firewall/firewall.tf to prevent forced replacement of the firewall transit public IP
- Add ip_tags to lifecycle { ignore_changes = [...] } on azurerm_public_ip.appgwpip in core/terraform/appgateway/appgateway.tf to prevent forced replacement of the application gateway public IP and the cascading recreation of dependent DNS zones
Provider/version updates
- Update the workspace base AzureRM provider pin from 4.57.0 to 4.73.0 (latest)
- Update the workspace base Terraform lock file to match the pinned AzureRM provider version
- Bump core/version.txt to 0.16.17 to satisfy CI version-change enforcement for the core Terraform changes
Template/versioning
- Bump tre-workspace-base to 2.8.4
- Update CHANGELOG.md to describe the AzureRM soft-delete-based fix and public IP lifecycle changes

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/c847d284-ba5e-406d-ba8d-c5c4eaa47d84 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

marrobi · 2026-05-21T16:34:07Z

@copilot no, dont use azapi:

Enabling soft delete does not prevent you from deleting the Backup Vault itself, as long as there are no protected (or soft‑deleted) backup items still in it.
What's actually happening
Azure now requires soft delete to be enabled on Microsoft.DataProtection/BackupVaults — the API rejects vaults created with soft_delete = "Off". That's why your azurerm_data_protection_backup_vault deploy is failing. The supported values are:
• AlwaysOn – soft delete enabled and immutable (cannot be turned off). ⚠️ This makes the vault very hard to remove.
• On – soft delete enabled (default) but can be disabled later. ✅ Safe choice.
• Off – no longer accepted by the platform for new vaults.

So set it explicitly to on. But consider side effects and need to maintain current functionality. Also ensure the azureRm provider is the latest release (pinned).

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/09dce1c2-3f0c-4c8a-b9bc-950d78b75e6d Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot · 2026-05-21T16:40:40Z

@copilot no, dont use azapi:

Enabling soft delete does not prevent you from deleting the Backup Vault itself, as long as there are no protected (or soft‑deleted) backup items still in it....

Addressed in 3d11658. The vault is back on AzureRM, soft_delete_enabled is set to true, the workspace base AzureRM pin/lock were updated to 4.58.0, and the provider now enables recovery_service.purge_protected_items_from_vault_on_destroy with non-retaining VM cleanup so workspace deletes keep the expected behavior.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint-api.hashicorp.com
- Triggering command: /tmp/terraform-bin/terraform /tmp/terraform-bin/terraform fmt -check -recursive (dns block)
- Triggering command: /tmp/terraform-bin/terraform /tmp/terraform-bin/terraform -chdir=templates/workspaces/base/terraform init -backend=false -upgrade (dns block)
- Triggering command: /tmp/terraform-bin/terraform /tmp/terraform-bin/terraform -chdir=templates/workspaces/base/terraform validate (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Copilot

Pull request overview

This PR updates the workspace base Terraform template to comply with Azure’s requirement that soft delete is enabled on newly created Recovery Services Vaults, while preserving current workspace deletion/cleanup behavior by ensuring protected items are purged on destroy.

Changes:

Enable soft_delete_enabled = true for the workspace backup azurerm_recovery_services_vault.
Configure AzureRM provider recovery_service features to purge protected items and avoid retaining VM backup data on destroy.
Bump the workspace base AzureRM provider pin/lockfile to 4.58.0 and update the template version + changelog entry.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
templates/workspaces/base/terraform/providers.tf	Bumps AzureRM to 4.58.0 and configures provider recovery services destroy behavior.
templates/workspaces/base/terraform/backup/backup.tf	Enables soft delete on the workspace backup Recovery Services Vault.
templates/workspaces/base/terraform/.terraform.lock.hcl	Updates AzureRM provider lock to 4.58.0 and corresponding hashes.
templates/workspaces/base/porter.yaml	Bumps `tre-workspace-base` bundle version to 2.8.4.
CHANGELOG.md	Documents the fix under BUG FIXES.

Files not reviewed (1)

templates/workspaces/base/terraform/.terraform.lock.hcl: Language not supported

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/fb2a7fcc-0cc4-47ff-acac-961ef1cb9334 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot · 2026-05-21T18:02:44Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

checkpoint-api.hashicorp.com
- Triggering command: ./terraform ./terraform version (dns block)
- Triggering command: /tmp/terraform /tmp/terraform providers lock -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64 -platform=windows_amd64 hashicorp/azurerm (dns block)
- Triggering command: /tmp/terraform /tmp/terraform fmt -check -recursive (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

marrobi · 2026-05-21T18:09:22Z

/test-extended 43c865c

github-actions · 2026-05-21T18:10:02Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/26244326090 (with refid 14403c22)

(in response to this comment from @marrobi)

Copilot · 2026-05-22T16:16:42Z

@copilot the lifecycle rule should not be on azurerm_recovery_service_vault as you've put, it should be on module.firewall.azurerm_public_ip.fwtransit. Here is what shows up in the pipeline:

Moved ip_tags to the correct resource — added it to the lifecycle { ignore_changes = [tags, zones, ip_tags] } block on azurerm_public_ip.fwtransit in core/terraform/firewall/firewall.tf, and removed the incorrect placement from the recovery services vault. Commit: the latest push.

rudolphjacksonm · 2026-05-22T17:28:13Z

@copilot bump API versions so the image can be built

Copilot · 2026-05-22T17:31:24Z

@copilot bump API versions so the image can be built

Bumped core/version.txt from 0.16.16 to 0.16.17 to resolve the CI "Code update without version change" failure caused by the core/terraform/firewall/firewall.tf change.

rudolphjacksonm · 2026-05-22T17:33:41Z

@copilot do we need to update the API version given this is a Terraform change? I'm not sure why the packer template version is getting updated in this PR

Copilot · 2026-05-22T17:36:15Z

@copilot do we need to update the API version given this is a Terraform change? I'm not sure why the packer template version is getting updated in this PR

Yes, the bump is required. core/version.txt is not a packer template version — it's the version number for the core Terraform/infrastructure component, listed by devops/scripts/list_versions.sh. The build validation CI enforces that any change under core/** must be accompanied by a change to core/version.txt (the "Stale version: core" check at line 79–82 of build_validation_develop.yml). Because core/terraform/firewall/firewall.tf was changed, the version bump to 0.16.17 is needed to pass CI.

rudolphjacksonm · 2026-05-22T17:47:31Z

@copilot do we need to update the API version given this is a Terraform change? I'm not sure why the packer template version is getting updated in this PR

Yes, the bump is required. core/version.txt is not a packer template version — it's the version number for the core Terraform/infrastructure component, listed by devops/scripts/list_versions.sh. The build validation CI enforces that any change under core/** must be accompanied by a change to core/version.txt (the "Stale version: core" check at line 79–82 of build_validation_develop.yml). Because core/terraform/firewall/firewall.tf was changed, the version bump to 0.16.17 is needed to pass CI.

@copilot that's fine, but I see a change has been made to porter.yaml as well and want to make sure this is also required

Copilot · 2026-05-22T17:53:19Z

@copilot do we need to update the API version given this is a Terraform change? I'm not sure why the packer template version is getting updated in this PR

Yes, the bump is required. `core/v...

Yes, the porter.yaml version bump (2.8.3 → 2.8.4) is required. Porter bundles are versioned — when the Terraform files inside templates/workspaces/base/terraform/ change, the bundle version must be bumped so that the resource processor can detect the change and deploy the updated bundle. Without this bump, an existing deployment would see the same version and not re-deploy with the updated Terraform configuration.

rudolphjacksonm · 2026-05-22T18:18:27Z

/test-extended 4dadc48

github-actions · 2026-05-22T18:19:00Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/26304650650 (with refid 14403c22)

(in response to this comment from @rudolphjacksonm)

rudolphjacksonm · 2026-05-22T20:04:13Z

/test-destroy-env

github-actions · 2026-05-22T20:07:10Z

Destroying PR test environment (RG: rg-tre14403c22)... (run: https://github.com/microsoft/AzureTRE/actions/runs/26309365126)

github-actions · 2026-05-22T20:07:12Z

Destroying branch test environment (RG: rg-tree5944c8a)... (run: https://github.com/microsoft/AzureTRE/actions/runs/26309365126)

github-actions · 2026-05-22T20:07:18Z

Branch test environment destroy complete (RG: rg-tree5944c8a)

marrobi · 2026-05-22T21:21:27Z

@rudolphjacksonm looks like this is maybe being casused by ip_tags on the ap gw pip. Needs ignore_changes too.

The dns zone derives the fqdn from it hence the cascade

resource "azurerm_private_dns_zone" "nexus" {
name = "nexus-${module.appgateway.app_gateway_fqdn}"
resource_group_name = azurerm_resource_group.core.name
...
}

@copilot add ignore_changes for the app gw pip ip_tags property.

… replacement Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/de009e2c-bd9e-4c65-a86b-a1939648a2e8 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot · 2026-05-22T21:23:36Z

@rudolphjacksonm looks like this is maybe being casused by ip_tags on the ap gw pip. Needs ignore_changes too.

The dns zone derives the fqdn from it hence the cascade...

Added ip_tags to the lifecycle { ignore_changes = [tags, zones, ip_tags] } block on azurerm_public_ip.appgwpip in core/terraform/appgateway/appgateway.tf in commit 713eedd.

marrobi · 2026-05-22T21:30:29Z

/test-extended 713eedd

github-actions · 2026-05-22T21:31:21Z

🤖 pr-bot 🤖

🏃 Running extended tests: https://github.com/microsoft/AzureTRE/actions/runs/26312937429 (with refid 14403c22)

(in response to this comment from @marrobi)

Initial plan

80e2afd

Copilot AI assigned Copilot and marrobi May 21, 2026

Copilot started work on behalf of marrobi May 21, 2026 15:08 View session

Copilot AI linked an issue May 21, 2026 that may be closed by this pull request

Workspace deployment fails on Recovery Service Vault - TF Provider issue. #4888

Open

Use AzAPI for workspace backup vault creation

cdd1bf7

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/c847d284-ba5e-406d-ba8d-c5c4eaa47d84 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix workspace deployment failure on Recovery Service Vault~~ Use AzAPI for workspace backup Recovery Services vault creation May 21, 2026

Copilot finished work on behalf of marrobi May 21, 2026 15:23

Copilot AI requested a review from marrobi May 21, 2026 15:23

Copilot started work on behalf of marrobi May 21, 2026 16:34 View session

Use AzureRM soft delete for workspace backup vault

3d11658

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/09dce1c2-3f0c-4c8a-b9bc-950d78b75e6d Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot AI changed the title ~~Use AzAPI for workspace backup Recovery Services vault creation~~ Enable soft delete for workspace backup Recovery Services vaults May 21, 2026

Copilot finished work on behalf of marrobi May 21, 2026 16:41

marrobi reviewed May 21, 2026

View reviewed changes

Comment thread templates/workspaces/base/terraform/providers.tf Outdated

marrobi requested a review from Copilot May 21, 2026 17:43

Copilot started reviewing on behalf of marrobi May 21, 2026 17:43 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Copilot started work on behalf of marrobi May 21, 2026 17:57 View session

Update AzureRM provider pin to 4.73.0 and update lock file hashes

43c865c

Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/fb2a7fcc-0cc4-47ff-acac-961ef1cb9334 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot finished work on behalf of marrobi May 21, 2026 18:02

Copilot AI requested a review from marrobi May 21, 2026 18:02

marrobi approved these changes May 21, 2026

View reviewed changes

marrobi requested review from ChrisChapman-gh and rudolphjacksonm May 21, 2026 18:11

Copilot started work on behalf of rudolphjacksonm May 21, 2026 18:58 View session

Copilot finished work on behalf of rudolphjacksonm May 22, 2026 16:17

Copilot started work on behalf of rudolphjacksonm May 22, 2026 17:28 View session

Bump core version to 0.16.17 for firewall ip_tags lifecycle change

4dadc48

Copilot finished work on behalf of rudolphjacksonm May 22, 2026 17:31

Copilot started work on behalf of rudolphjacksonm May 22, 2026 17:33 View session

Copilot finished work on behalf of rudolphjacksonm May 22, 2026 17:36

Copilot started work on behalf of rudolphjacksonm May 22, 2026 17:47 View session

Copilot finished work on behalf of rudolphjacksonm May 22, 2026 17:53

Copilot started work on behalf of marrobi May 22, 2026 21:21 View session

Add ip_tags to ignore_changes on appgwpip public IP to prevent forced…

713eedd

… replacement Agent-Logs-Url: https://github.com/microsoft/AzureTRE/sessions/de009e2c-bd9e-4c65-a86b-a1939648a2e8 Co-authored-by: marrobi <17089773+marrobi@users.noreply.github.com>

Copilot AI changed the title ~~Enable soft delete for workspace backup Recovery Services vaults~~ Enable soft delete for workspace backup Recovery Services vaults and fix core public IP lifecycle issues May 22, 2026

Copilot finished work on behalf of marrobi May 22, 2026 21:24

Copilot AI requested a review from marrobi May 22, 2026 21:24

Conversation

Copilot AI commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is being addressed

How is this addressed

Uh oh!

marrobi commented May 21, 2026

Uh oh!

Copilot AI commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI commented May 21, 2026

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

marrobi commented May 21, 2026

Uh oh!

github-actions Bot commented May 21, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

rudolphjacksonm commented May 22, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

rudolphjacksonm commented May 22, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

rudolphjacksonm commented May 22, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

rudolphjacksonm commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

Uh oh!

rudolphjacksonm commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

Uh oh!

marrobi commented May 22, 2026

Uh oh!

Copilot AI commented May 22, 2026

Uh oh!

marrobi commented May 22, 2026

Uh oh!

github-actions Bot commented May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented May 21, 2026 •

edited

Loading

Copilot AI commented May 21, 2026 •

edited

Loading